PREDICTING THE CLASS OF WINE WITH NEURAL NETWORKS

An example of a multivariate data type classification problem using Neuroph

by Ana Nikolic, Faculty of Organisation Sciences, University of Belgrade

an experiment for Intelligent Systems course

Introduction

The goal of this experiment is to show how neural networks and Neuroph Studio are used to resolve problems of classification. We will show that with creating different architectures for solving the problem, we can found the best solution.

A classification process involves assigning objects into predefined groups or classes based on a number of observed attributes related to those objects. Although there are some more traditional tools for classification, such as certain statistical procedures, neural networks have shown to be an effective solution for this type of problems. There is a number of advantages for using neural networks - they are data driven, they are self-adaptive, they can approximate any function - linear as well as non-linear (which is quite important in this case because groups often cannot be divided by linear functions). Neural networks classify objects rather simply - they take data as input, derive rules based on those data, and make decisons.

For better understanding of our experiment, we suggest that you first look at the links below:

Neuroph Studio-Geting started

Multi Layer Perceptron

Introducing the problem

Our asigment is to train neural networks to predict witch type belongs the wine, when it is given other attributes as input. First thing we need is a data set. We’ll use a data set found on http://archive.ics.uci.edu/ml/ . The name of the data set is Wine Data Set (1991-07-01). These data are the results of a chemical analysis of wines grown in the same region in Italy but derived from three different cultivars. The analysis determined the quantities of 13 constituents found in each of the three types of wines. The data set contains 178 instances, and 13 attributes.

 The input attributes are:

1) Alcohol 
2) Malic acid 
3) Ash 
4) Alcalinity of ash 
5) Magnesium 
6) Total phenols 
7) Flavanoids 
8) Nonflavanoid phenols 
9) Proanthocyanins 
10) Color intensity 
11) Hue 
12) OD280/OD315 of diluted wines 
13) Proline 
 Each instance has one of 3 possible classes( tree types of wine).

The output attributes are:

1) 1 (first type od wine)
2) 2 (second type of wine)
3) 3 (third type of wine)

The data set can be dowloaded here, but if we want to insert it in Neuroph we need to prepare the data first. The type of neural network that will be used in this experiment is multi layer perceptron with backpropagation.

Download the data set

In order to train a neural network, there are five steps to be made:

1. Preparing the data
2. Create a Neuroph project
3. Create a training set
4. Create a neural network
5. Train the network
6. Test the network to make sure that it is trained properly

  First thing we need to do before creating our network is to prepare data for Neuroph to understand them. Because in our data set we have different data types we wil first normalize integer type of data.

We're using the standard Min Max normalization formula:

The first attribute of the class l is not used this method of normalization because the attribute takes the value 1, 2 or 3 and more appropriate method would be to turn the three classes into three outputs. If an instance belongs to the first class, the first class of wine, the value of the first output will be 1, and of the second output 0 and the value of the third output will be 0. If an instance belongs to the second class, the second type of wine, the value of the second output for that instance will be 1, and the value of the first and third output will be 0. Finaly, if an instance belongs to the third class, the third type of wine, the value od the third output will be 1, and the value if the first and the second ouyput will be 0.That way, all three outputs will be values 0 or 1, which fits in our model where all the data are in the 0-1 range.

Now we need to  create a new training set. First, we need to create a new Neuroph projects by clicking on the 'File' menu, and then 'New project'.

We named the project “NeurophWineProject”.

The final step is to click 'Finish'  button, and new project is created. The project will be shown in the top left corner of Neuroph Studio.

To create a new training set, we must do the right-clicking on our project, and choose option 'New', and than 'Training set'. We must name it, set the parameters and choose option  'Supervised' training , because we want to minimize the error of prediction through an iterative procedure. Supervised training is accomplished by giving the neural network a set of sample data along with the anticipated outputs from each of these samples. That sample data will be our data set. Supervised training is the most common way of neural network training. As supervised training proceeds, the neural network is taken through a number of iterations, until the output of the neural network matches the anticipated output, with a reasonably small rate of error. Error rate we find to be appropriate to make the network well trained is set just before the training starts. Usually, that number will be around 0.01.
In our case number of inputs will be 13, because we have 13 input attributes, and the number of outputs will be 3, because we have 3 output attributes.

Now we need to click next, and than edit the training set table. We click a 'Load from file' and select a file where is the data set from our computer. We will use 70% data form our original data set for training. We also need to select a values separator. In this case, it is tab. After finishing these steps out table will be loaded and will appear.

This is how our table looks after loading.

After finishing this, we need to start creating neural networks. We will create several neural networks, but with different sets of parameters, and determine which is the best solution for our problem by testing them. That means that we will have more training attempts.

We will create it by right-clicking our project in the 'Projects' window, and then clicking 'New' and 'Neural Network'. We will set the name and the type of the network.We will name it ‘NewNetwork1’ and choose Multi Layer Perceptron. Multi layer perceptron is the most widely studied and used neural network classifier. It is capable of modeling complex functions, it is robust (good at ignoring irrelevant inputs and noise) ,and can adapt its weights and/or topology in response to environment changes. Another reason we will use this type of perceptron is simply because it is very easy to use - it implements black-box point of view, and can be used with few knowledge about the relationship of the function to be modeled.

Now we can click 'Next’ and than set parameters that we need for multi layer perceptron. The number of input and output neuron is the same as the number of inputs and outputs in the training set.  Now we have to select the number of hidden layers, and the number of neurons in each layer. We will choose  only one layer, and for the first training attempt we will choose three hidden neurons.

We have checked 'Use Bias Neurons', and chosen sigmoid transfer function (because the range of our data is 0-1, had it been -1 to 1, we would check 'Tanh'). As a learning rule we have chosen 'Backpropagation with Momentum'. This learning rule will be used in all the networks we create, because backpropagation is most commonly used technique and is most suited for this type of problem. In this method, the objects in the training set are given to the network one by one in random order and the regression coefficients are updated each time in order to make the current prediction error as small as it can be. This process continues until convergence of the regression coefficients. Also, we have chosen to add an extra term, momentum, to the standard backpropagation formulae in order to improve the efficiency of the algorithm.
When we click 'Finish', and the first neural network which we will test is completed.

If you want to see neural network as a graph, just select 'Graph View'. Right nodes in first and second level are bias neurons.

For training the neural network we’ll use the other 30% of the original data set. We select the training set and click 'Train'. In a new window that opens, we set the learning parameters. The maximum error will also be 0.01, and learning rate 0.2 and momentum will be 0.7. Than we click 'Train' .

The graph went down after 45 iterations.

The test showed that total mean square is 0.04236700941165338. The goal of experiment is to construct experiments in such a way that when the observations are analyzed, the mean square error is close to zero relative to the magnitude of at least one of the estimated treatment effects.

Now we will test this network with several input values to see how the network will behave. We will select 5 random input values from our data set.

Those are:

We can see that the neural network guessed only three of five instance, and even the Total Mean Square Error is acceptable, we will continue with finding a better solution.

In the second attempt we will create a new neural network, called ‘NewNetwork2’ and the number of hidden neurons will now be 8.

The neural network graph looks like this:

For training the neural network we’ll also use the other 30% of the original data set. We select the training set and click 'Train'. In a new window that opens, we will set the learning parameters. The maximum error will be 0.01, and learning rate 0.6 and momentum will be 0.9. Than we click 'Train' .

The graph went down after 36 iterations  and we can conclude that network quickly learned the data set.

After testing the network we can see that the Total Mean Square Error is 0.030780706500805676, less than before, and conclude that this network is better solution than the previous one.

Like before, we will see how network react with the 5 inputs we random chose .

The network guessed correct in all five instances. After this test, we can conclude that this solution does not need to be rejected.

In the next attempt we will create a neural network, called ‘NewNetwork4’ and we will set the number of hidden neurons to 11 .

The neural network graph looks like this:

For training the neural network we will select the training set and click 'Train'. In a new window that opens, we will set the learning parameters. The maximum error will be 0.01, and learning rate 0.2 and momentum will be 0.7. Than we click 'Train' .

The graph went down already after 34 iterations  and we can conclude that network very quickly learned the data set.

The Total Mean Square Error now is 0.030780706500805676, and we can say that this network is not a better solution than the previous one.

Like before, we will see how network react with the 5 inputs we random chose .

Again, the network guessed all five instances, and the solution is acceptable.

We will train again the previous  neural network that we created, called 'NewNetvork4' but we will change maximum error, learning rate and momentum to 0.01, 0.6, 0.9. Than, we click 'Train' and waiting to see what happens.

We can see in the picture that train was successful and that network learned the exemple after 70 iterations.

Total Mean Square Error after testing NewTest1 training set, is 0.021718430058843522. It is also a small and  acceptable error.
We can see that the same network but with the different parametars give the better result.

We will now try to re-train the same network, but only decrease the learning rate on 0.3. Momentum remains the same, 0.9, as in the previous session.

We can see the big difference betwean the last two trainings. Now the network learn the exemple quicklier, after 40 iterations.

And the Total Mean Square Error is little bigger than before,  but still acceptable, 0.03183341152026723.

Following these rules, we now decide for a neural network that contains 10 hidden neurons in one hidden layer. Again, we type in the standard number of inputs and outputs, check 'Use Bias Neurons', choose a Sigmoid Transfer function, and select 'Backpropagation with Momentum' as the Learning rule.

Now, we will choose ten hidden neurons.

Graphical representation of neural network looks like this:

Like the previous neural network, we will train this one with the training set we created before, with the entire sample. We select 'NewTest1', click 'Train' and a new window appears, asking us to fill in the parameters. We will set the maximum error to be 0.01, the learning rate will be 0.4, and momentum 0.7. After we click 'Train', the iteration process starts. We can see on the picture that the iteration process stops at 35 iteration.

This time, we didn’t get the better result than before, and the Total Mean Square Error now is 0.05860080947293679.

Even the the Total Mean Square Error isn’t the best we got we’ll test network with 5 random inputs we chose before.

The network guessed correct four of five instances. After this test, we can conclude that this solution does not need to be rejected.

In order to reduce some errors we will re-train the same network but with different training parametars. We will set maximum error, learning rate and momentum to 0.01, 0.2, 0.7 to see what happens.

We can see in the picture that train was successful and that network learned the exemple after 36 iterations.

This time, we got better result than before, and the Total Mean Square Error now is 0.02841794491082484.

In this attempt we will try to use 9 hidden neuones in one hidden layer to see how the network will react in order to reduce errors as much as possible. First, we will create a new network called NewNetwork6. When we click 'Finish', the neural network is completed. If you want to see neural network as a graph, just select 'Graph View'.

We will select the training set and after clicking 'Train', we will set the learning parameters. The maximum error will be 0.02, and learning rate 0.2 and momentum will be 0.7.

The graph went down to the horizontal asymptote, immediately after 33 iterations. This means that the neural network quickly learned 70 percent of date set.

 The goal of experiment is to construct experiments in such a way that the mean square error is close to zero relative to the magnitude of at least one of the estimated treatment effects.With this network Total Mean Square Error is 0.02676905451038841 and this is the best result we got.

The final part of testing this network is testing it with several input values.

We can conclude that network guessed correct in all five instances.

Conclusion

As a conclusion we will show a table that summarizes all the training attempts and we will see witch the best solutions for this problem are. We can see form the table that the number of hidden neurons is crucial to the effectiveness of a neural network. Also, the experiment showed that the success of a neural network is very sensitive to parameters chosen in the training process. The learning rate must not be too high, and the maximum error must not be too low. The results have shown that the total mean square error does not reflect directly the success of a network.

• Original training set

• Normalised training set

• Training set page at UCI




• Neuroph Studio 2.7 project

• Neural network training in Java source (NetBeans project with Neuroph 2.7)

See also:
Multi Layer Perceptron Tutorial